DEGENERATE PRINCIPAL SERIES REPRESENTATIONS OF 

SO(p + l,p) 



V. FISCHER AND G. ZHANG 



Abstract. For p odd, the Lie group G" = SO (p + l,p+ 1) has a family of 
unitary degenerate principal series representations realized on the space of real 
(p + 1) X (p + 1) skew symmetric matrices, similar to the Stein's complemen- 
tary series for SL(2n,C) or Spch's representation for SL(2n,R). We consider 
their restriction on the subgroup Go = SOo(p + l,p) and prove that they are 
still irreducible and is equivalent to (a unitarization of) the principal series 
representation of G, and also irreducible under a maximal parabolic subgroup 
of G. 



1. Introduction 

In the present paper we shall study the unitarity of degenerate principal se- 
ries representations of the group G = SO(p + l,p) induced from certain maximal 
parabolic subgroup for odd p = 2q — 1. 

In the case of the group SOo(«, n), or more generally SU(n, n; F), for F = IR, C, H, 
Johnson [15) has determined the range of unitarity of the representations; for 
SU(n, n, F), n even, he found certain complementary series. Some generalizations 
of these results were obtained in [20l HH [28l [12] for a larger class of groups by using 
computations in the compact picture. The analysis of these representations in the 
non-compact picture has been done in [2]. 

We shall prove that the restriction of the complementary series of G" = SOo (p + 
l,p+ 1) to the opposite maximal parabolic subgroup of G is irreducible, and in 
particular the restriction to the identity component Go of G is irreducible. We shall 
use mostly the non-compact realization of the principal series. The proof relies on 
both Euclidean and nilpotent Fourier transform. 

The restriction of the degenerate principal series representations of SO(n,n) to 
the subgroups SO(n, m) x SO (n — m) for m < n has been studied earlier by Lee-Loke 
[17] in the compact picture, the representations of SO(n, m) x SO(n — m) appearing 
are of the form r x r', and the representations r are degenerate principal series. 
It might be true that the representations r of SO(n,m) are also irreducible under 
corresponding the maximal parabolic subgroup, as we show here for m = n — 1. 
We mention also that there has been quite some study of complementary series 
representations for semisimple Lie groups. In [1 a large class of complementary 
series representations is constructed with parabolic subgroups being cuspidal and 
maximal; our case of SO(p + l,p) here is however not cuspidal. For the groups 
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SO(p, q) some complementary series similar to ours can be constructed by using the 
branching of holomorphic representations of SU(p, q) to SO(p, q). However those 
constructions works only for q — p > 2 (see [HI Therorem 3.1], Therorem 5.2]), 
and thus they do not cover our present case of SO(p + l,p). Our result about the 
irreducibility of the restriction to SO(p+l,p) of representations of SOo(p+l,p+l) is 
in a way similar to the Kirillov conjecture, now a theorem for GL p _|_i(R) and 

GLp(K) x M. p . See also [22] on the study of the restriction of complementary series 
of SO (rt, 1) to SO(n — 1, 1), [T3] on branching of highest weight representations, and 
[H] the classification of finitely decomposable representations of Cr under G. We 
note also that the irreducibility result under the the parabolic group P can possibly 
be also proved abstractly by using the Mackey theory on induced representations. 
However we present a relatively elementary and direct proof, in particular it also 
yields a decomposition of the representation under the subgroup NSp(q — 1,R). 

2. Preliminaries 

2.1. The group G» = SO (p + 1, j> + 1) and G = SO(p+ l,p). 

Let M Pi9 be the space of real p x q-matrices, and denote M p = M PiP . Denote 
X = X p = {X G M p ;X — ~ X 1 } the subspace of skew-symmetric real matrices. We 
will also use the short-hand notation X~ l = (X 1 ) , for an invertible X. Denote 
I n the identity matrix in M„ and I Ptq — diag(— I pj I q ). 

Let p > 1 and G" = SOo(p+ 1) be the identity component of SO(p + l,p + 
1) = {g £ M 2p+2 ;dct5 = 1, g l I p+ i^ + ig = I p+ltP+1 }, and G = SO(p+l,p) realized 
as the subgroup of G" via: 

G = {diag( 5 ,l)GG»}cG«. 

The group G has two connected components and we denote its identity component 
by G . 

Elements g of G" will be written as 2 x 2 block matrices 

(91,1 9iA 

\92,l 92,2 J 

with each entry being in M p +i. The Lie algebra g" of G" has the decomposition 
fl" = t' © p" with t' = so(p + 1) © so(p + 1) with respect to the Cartan involution 
g — > g~ l . The group 

= {diag(fci,fe); ki,k 2 G SO(p + l)} = SO(p + l) x SO(p + l) , 

is a maximal compact subgroup of G" with Lie algebra t' . Correspondingly q = t®p, 
6 = so(p + 1) ©so(p) and 

K Q = {diag(fc 1 ,fc 2 ,l); fciSSOO+1), fc 2 G SO(p)} - SO(p + 1) x SO(p) 

is a maximal compact subgroup of Go, while 

X - = {diag(*i J fc 2 ,detA!2); fciGSO(p + l), i 2 eO(p)}~SO(p + l)xO( P ) 

is a maximal compact subgroup of G. Note that X = {/ p +i, p +i, / 2p + 2 } X Kq ~ 
Z 2 x P. 

For j = 1, • • • ,p + 1, let 

ff i=(£ whcrc X = diag(0,-.-, 0,1,0,-. -,0), 
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with 1 on the jth position. Then := RHi H hKH p+ i and t := RHi H \-RH p 

are maximal abelian subspaces of p" and p. Let {e^} be the dual basis of {Hj}. 
The positive root systems of (g*, t") and (g, t) are {ej ± Ck, l<j<k<p+l} and 
{ej ±e k , 1 < j < k < p}. 

2.2. The maximal parabolic subgroups P" and P. 

We fix the elements: 

= Hi + • • • + H p+1 ep» and £ = ^ + • • • + F p G p . 

Let a* = R^, a = R£. The root space decomposition of g* under £" and g under £ 
is then 

g 1 * = nl 2 + m 8 + a" + n 2 , g = n_ 2 + n_i + m + a + tii + n 2 , 

with roots ±2, 0, and ±2, ±1, respectively. We set for the corresponding positive 
root subspaces: 

n" = n 2 and n = tti + n 2 , 
and for the negative root spaces: 

it* 1 := n tt _ 2 = (n 2 )* and n := n_i © n_ 2 = tt* . 
We shall use explicit forms for the root spaces: 

n" = {n\, Z 6 X p +i} ~ where := 

and 

(z v — z 
—v l i>* 
Z H — 2 

We have: 

n = tii © n 2 with m = n , R P - W , n 2 = n^.o ~ • 

The Lie algebra n" is abelian and n is a 2-step nilpotent Lie sub-algebra of g. 
Elements ri( z v ) will simply be written as (z,v). The Lie bracket in n is given, via 
the above identification it = X p © W, is 

[zi +vi,z 2 + v 2 ] = vivl - v 2 v\. 

Thus n is the free nilpotent Lie algebra with p generators (over R) . 
The centralizer of a" in jj" is 

m» © a J = {l\ XtYy X = X\ Y* = -Y G M p+1 }, J; 

identified with gl(p + 1) via $ XY ) ^ X + Y G + 1), whereas the centralizer of 
a in g is 

/?/ x\ 

m©a = {diag(Z (Xi!/) ,0); a; = x t ,y t = -y G M p }, J (x>!/) := 01, 

\x y/ 

identified with gl(p). Note that 

(2.1) gl(p)~m©a C m" © a J ~ g[(p + 1) . 





Vx y)> 
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Let M", A$, iV* and iV" be the simply connected sub group of G" with Lie al- 
gebras m s , a B , n" and n B respectively. Let P" = MtUW and P J = M^SJV 8 be 
the corresponding maximal parabolic subgroups of G" . Similarly we define the con- 
nected subgroups Po = M AN and Po = M AN of Go with Lie algebra m + a + n 
and m + a + fi. Note that the centralizer of a" in is 

Z Ki {^) =K»nA/» = {diag(fci,fci); h e SO(p+l)} ~SO(p+l) , 

and the centralizer of a in K is 

Zjc(a) = K n M = {diag(fc 2 , det fc 2 , fc 2 , det fc 2 ); fc 2 e 0(p)} - 0(p) , 

and the centralizer of a in i£o is the connected component of the identity of Zk{&)- 
We set: 

M = Z K {a)M , P = Z K (a)P = MAN and P = Z K (a)P = MAN . 

The group M*A" is isomorphic to the matrix group GLp +1 = {/i e M p+1 , det /i > 
0} via: 

(2.2) h e GL+ —> fc< , ° t ) fc D G M«A« , fc D = 2-4 ^ ' 1 



>+l h -t J-O^ , - - ^ 

Restricting this isomorphism to {diag(/i, det h), h £ GL p } and to {diag(/i, det h), h £ 
GLp }, we obtain an isomorphism between MA and GL P and between MqA and 
GL^ . Thus, using the isomorphisms just above as identifications, P", P and Po 
can be described as the semi-direct product: 

P B = GL+ +1 7V» , p = GLpiV and P = GL+7V . 

Lemma 2.1. The following inclusions of Lie algebras hold: 

ra © a C m" © a B anc! m © a © n C m tt © a tt © n tt . 

We have: 

M A C MA C Af B A» and P C P C P* . 
Moreover we have a factorization o/exp(7i( Zi „)) £ N in GL^" +1 = Af"A", 

(2.3) expn (Z: , ) =mexpTi^ ((z ii) , 

where rn € Af" corresponds to ^"j £ SL p+ i and 

6 <*2 g , 



(2.4) M(z,v):= 
is viewed as an element of n" . 

Proof. The first relation is in (|2.1[) . We can write ?i( Z)t) ) = ijLr k n + n^ w with: 

This shows n c m" © and implies m ffi a © n C m" © a" © n". The group inclusions 
follow immediately. Easy matrix computations give the equality (12.3[) . □ 

Note that the adjoint action of MA = GL^ and GL P on N is 

(2.5) g ■ (z,v) = (gzg\gv) . 
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3. Degenerate principal series representations 
Throughout the paper we assume p is odd and we write 2q = p + 1 . 

3.1. Principal series of G» = SO (p + l,p + 1). 

Let /i £ C and consider the induced representation P (/j,) of G* from the following 
character on P* 

xl : met** n i— > e (t * +p>)t , 

where p" = q(2q — 1) is the half sum of the positive root of ad£". In terms of 
P» = GL+ AT" this is 

(3.1) x»:in.— ►det(i)^, I G GL+ . 

This representation can be realized on the space of Haar measurable functions 
f(g) on G" such that 

(3.2) f(gln) = det^)"^ /(,?), ln£P\ 
and 

(3.3) /|^gl 2 (a:«). 

See [IB]. The group G» acts on P(n) by the left regular action and we denote the 
representation by (/"(//), 7r^). The condition (|3.2p implies that / £ P(fJ>) is invariant 
under the right action of n M " and can therefore be identified as functions on 
Kt/K*r\MK However K^/K^ n M s can be realized as SO(2g) since the group if" 
acts on SO(2g) by 

. ,.,.,,( SO(2g) — ► SO(2g) 

and the isotropy group of the identity matrix £ SO(2g) is if" n M". Thus 
condition (|3.3p can be equivalently replaced by 

f\ Kt/KtnMt G L 2 (#V#» DM*) = L 2 (SO(2«)) . 

We denote by (I Kt (fi), tt^) the space of if "-finite elements. We will need its 
decomposition under if". Recall, see e.g. |15j . that each irreducible representation 
of SO(2g) is determined by a (/-tuple of integers: 

m = (mi, ■ • • , m q ), mi > •• • > m g _i > \m q \ . 

We write Vm for the representation space of m. Thus jjL, (/x) is the same as the 
space L 2 (SO(2q)) x „ of if ^-finite elements in L 2 (SO(2q)) and we have 1151 : 

(3.4) /^( M )~ J L 2 (SO(2g)) K » =^V a ®V4, 

m 

where V m stands for the dual representation of Vm- 

Each function in the space P (//) is also uniquely determined by its restriction to 
NK The G"-action in this realization is referred as ^"-realization. It is jTBl p. 169] 
the space L 2 (iV", e 2 ^(M+P*) H> ^ where the function ii" is denned by t — ii"(n) using 
the Iwasawa decomposition of n = kme^ n+ £ K^P*. Note however that the L 2 - 
norm in L 2 {K) or L 2 (7V", e 2S ^ +p ' , ^ ^r ' , ) is G"-invariant only for purely imaginary /x, 
H £ iR. 
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3.2. Zeta distribution and complementary series &{v) of GK 

The unitarity of (Z"(/i), ir^) for /i outside the standard unitary range /i G iR has 
been completely determined in [T5] in the if -finite realization; see also [301 HE] ■ Let 
/i G (0, q). There exists a gMnvariant (i.e. g" acts as skew Hermitian operators) 
positive definite inner product (•, •)„ on the space /^(m) of if "-finite elements of 
the principal series of I^{fJ.) (see [HI Theorem 7.5] and [H section 7]). We will 
denote the unitary representation of G" on the completion of /L, (/i) as (C"(/i), 77^). 

The non-compact picture has been further studied in [5] and we recall it briefly 
here. We identify TV* with X2 q via Z i-> expn^ an d we consider the following 
(formally defined) linear form on the Schwarz space S{X2 q ) 

f 7rf(s+2g-i) 
Z s {h)= ls ,2 q h{x)\Vi{x)\ s dx, lsM 



for s G C with sufficiently large Sfts where Pf denotes the Pfaffian polynomial. This 
defines [5l [2] a family of tempered distributions {Z s } which admits a holomorphic 
continuation to the whole complex plane and whose Fourier transform satisfies the 
following functional equality: 

(3.5) Z 2 g iS _( 2 q_i) (Fh) = Z 2q ,- S {h), s G C ; 
here the Fourier transform on X2 q is given by: 

J7i(C) = / h(x)e 27ri( - xX) dx, h e S(X 2q ) , 
J Mil 

The inner product in the ./V" -realization is given by: 

(/, := (/, Z s * g) LHX) = Z s (/ * .g) with a = ^ - (2q - 1) , 

where g : x M> /(—&), initially defined on the Schwarz space ^(A^g). From (|3.5p . 
we see that (/, /) M = Z_*i (| J 7 /! 2 ) for any / G S(X2 q ); this makes sense because for 
/i G (0, q), Z_t. is a locally integrable function. The completion of S(X2 q ) is 

(3.6) C«( M )={/6S'(^); |Pfr^/G£ 2 (* 29 )}, 

(Note that the condition / G S'(X2 q ) can be deduced also from iPfp^J 7 / g 
i 2 (A- 29 ).) 

We summarize the results in the following 

Proposition 3.1. Suppose /i G (0,g). There exists a ({j" ,7r") -invariant positive def- 
inite inner product (•, -)p on ZjL, (/x). /is completion defines a unitary representation 
of GK In the non-compact realization the Hilbert space is described by 



The operator 

(3.7) T^. f eS(X 2q )^0 = T- 1 (\Pf\-%Tf) eL 2 (X 2q ), 

then extends to a unitary operator from C'(fi) onto l?[X.2 q ). We denote the corre- 
sponding representation on L 2 (X2 q ) by 

(3.8) 7T» = .7>« J"" 1 . 

We obtain a simple description of the action of (P",ff") (see [2J sections 7 and 8]): 

(3.9) ^(exp 4)0(140 = 0(W - Z) 
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and for g — me*^ £ M^A^ identified with an element of GL+ +1 : 

(3.10) n^(W) = e-^WWg) . 

3.3. Principal series I(v) of G. 

Let /(^) be the principal series induced from the following character of P = 
GLpTV: 

Xu'.ln\ — > Idet^)]^ , 

where p = p 2 /2 is the half sum of the positive roots of ad£, with the similar 
condition as in (13.21) and (I3.3j) . That is, the representation space is realized as Haar 
measurable functions f(g) on G satisfying 

(3.11) /(<7Zn) = |det(0r^/(<?) , 
and 

(3.12) f\ K eL 2 {K) or equivalent^ f\ R E L 2 (K/K n M) ; 

the group G acts on I(v) by the left regular action and we denote this action by 
(I(v),tt u ). We denote by {Ik{v),tt v ) the space of if-fmite elements. 

The homogeneous space K/KnM can be realized as the Stiefel manifold of rank 
p isometries: 

K/K nl= V p+hp = {x e M p+1>p ; x'x = I p } , 
where the group K acts transitively via: 

Vp+i,p — > ^p+i,p 



K 3 diag(fci, kz) 



and K n M is the isotropy group of {^^j e Vp+i,p- The elements / in I(v) then 
satisfy 

/Ik/xhm e i 2 (W n M) - L 2 (V P+1>P ). 

We will need the multiplicity free decomposition of L 2 (V p+ i iP ) under K = 
SO(p + 1) x SO(p). Let us recall that each irreducible representation of SO(p) 
is determined by a (q — l)-tuples of integers 

n = (m, • • • , n q -i), ni> ■■■> n q -i > n g _i > , 

and we write Wn for the representation space. Given a representation m of SO(p+l) 
we write m ^ n if n appears in the irreducible decomposition of m under SO(p). It 
is a classical result, see e.g. |T0] [2§J [13] that n appears in m multiplicity free. This 
implies that the space of ifo-fmite elements of L 2 (V p +i tP ) is decomposed under K$ 
as follows: 

(3.13) I K {v)~L 2 {V p+1 , p ) Ko = J2 V m® W H' 

( in . ii ) : in ~ > ii 

and this decomposition is multiplicity free. 
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3.4. The restriction map P. 

We shall consider simply the restriction of functions in /"(//) to G C G'. To 
clarify its definition we note first that the space ZL , (/i) of ZC"-fiiiite functions are 
smooth functions on G* . Thus the restriction map 

i?:4M^C°°(G), Rf(g) = f(g),g&G 

makes sense. In the ZT"-realization of Z"(/Lt), we have Rf £ I?(K) for any Zl^-finitc 
elements / e L 2 (SO(2g)). 

Our main observation is the following: 

Proposition 3.2. Let v 1 /i G C such that v = ^jM- The restriction map R is a 
G-equivariant isomorphism from I Kt (fi) onto Ik{v) in the sense that 

R*Kg)f = ir u (g)Rf, f e P(p), geG, 

and it is unitary as a map from ZjL,(/x) ~ Z 2 (SO(2g))^« onto Ik (v) ~ Z 2 (V^,+i iP )i<- . 

Proof. Let / e ZjLj(/x). By Lemma T2.ll one check easily for In 6 P C P" XuO rl ) = 
Xv(ln). Together with (|3.2p . it implies that P/ satisfies p. lip . Moreover Q3.3P 
implies (13321 . So P/ s Z(i/) and Ptt» = n v (g)Rf for any geG. As / is 
ZC"-finite, P/ is also ZT-finite. 

The decompositions (13. 13)) and (|3.4I) show the rest of the claim. □ 

Using Proposition 13.11 we get that restriction to G of the complementary series 
C"(/i) defines a unitarizable representation of G, which we write as C(y), whose 
ZC-finite elements are the same as Zr-(V), by Proposition ^. 2[ i.e, 

C(v) = PC tt (^), C K (v) = I K {lj), v = -^—[i 1 (j,£(Q,q). 

p+1 

The main result of this paper is the following theorem which states that restric- 
tion C(V) to the maximal parabolic subgroup P of G is irreducible. 

Theorem 3.3. Let v = ^pjA* ™^ M £ (0, ?)• ZTien restriction to G of the com- 
plementary series (C"(/x), G") defines a unitarizable irreducible representation C{y) . 
It is the unitarization of the principal series representation (Ik (v) , G) realized in 
the non-compact picture. Moreover, its remains irreducible when restricted to the 
maximal parabolic subgroup GL P N = P. 

The irreducibility under Go in the above statement is essentially proved in [17] . 
Indeed let K = Spin(p + 1) x Spin(p). The representation C(v) is treated as rep- 
resentation of Spin(p + l,p) and it is proved [TTl 12.2.1] that the (q, ZT)-module of 
C(v) is irreducible. However the representations of K in C(v) descend to the rep- 
resentation of K and thus the (g, ZT)-module is the same as (g, Zl )-module Ck {v), 
and the latter is then irreducible. 

To prove the rest of Theorem 13.31 we will use the non-compact picture. As 
P C P B we can find the action of P on (tt» , L 2 (X 2q )): 

Lemma 3.4. The representation of P on (M, L 2 (X2 q )) is unitarily equivalent to 
the representation (tt,L 2 (N p )) given by: 

(3.14) 7r(no) • 4>{n) = ^(fij^n) 
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for an element hq 6 N and for an element g £ GL P : 
(3.15) 7r( 5 )0(n) = |dct 5 |-50( 3 - 1 .n) , 

where the action o/GL p on N is given by (2.5\) . 
Proof. Let us consider the unitary isomorphism: 

L2[Np l ^ ^ {X2q) givenby ^(z, v) = <f>(M(z, v)) . 
It is easy to check (|3.15p . Now for (z ,v ) s N p , using (|2.3|) . we compute: 

{^V™M{z ,v ) m h) • 4{M{z,v)) = (f>(-M{z ,v ) + h t M(z,v)h) 
and direct computations show 

-M{z , v ) + h l M (z, v)h = M ((z , v,,)- 1 ^, «)) 
so the action of N is given by (|3.14|) . □ 



So by Schur's Lemma, Theorem 13.31 is proved once we have shown the following 
proposition: 

Proposition 3.5. Let T be a bounded operator on L 2 (N) commuting with the 
action n of P defined in Lemma\3.4\ Then T is the scalar multiple of the identity. 



Remark: Using the non-compact pictures, one can show the unitarity of C(y). 
Indeed using Lemma l2.1l it is not difficult to show that the Knapp-Stein intertwiner 
A$ and A\ for the series fl(fi) and I{y) satisfies: 

(3.16) Al(f*)(expnl {z v) ) = A u (ft\ G )(expn (Z)V) ) , 

and the properties of Aj^ described in [2] imply that l v is unitarizable. 

Knapp-Stein intertwiners are (nilpotent) convolution operator with very singular 
kernels. By [2], A^ is an abelian convolution with a power of the Pfafnan. It can 
be computed using geometric means that the kernel of A v is of the form Q(z, v) s = 
det(z + iw') s for (z,v) € N and it is only through some elementary but tricky 
matrix computations that it can be linked with the abelian convolution with some 
power of the Pfaffian as in p,16[) . 

4. Proof of Proposition 13.51 

We will use some well-known results for the Plancherel formula and von Neumann 
algebras of left regular representations of N on L 2 (N); see [27l Chapt.14] and [7] 
for general locally compact groups [S] for the case of nilpotent groups. We describe 
first the support of the Plancherel measure described in [H [SI [21] . 

4.1. The support of the Plancherel measure. To ease notation we write ele- 
ments in n or N as n instead of n. We may identify them with elements of X2q by 
(I2.4j) : on Xi q we consider the standard inner product (Z,W) = ^TiZW*. Hence 
we have equipped n with an inner product and we can now identify the dual n* 
with n. The dual action of g 6 GL P on n* = n will be written as g * n. 

We fix a generic point o n * = (z 0n , , v 0n , ), the element of n* ~ n defined using 
(ED by: 

M(o„«) = J q where J q = diag( J, . . . , J) e Mi q and J=(®^ . 



10 



V. FISCHER AND G. ZHANG 



It is easy to see that a representative of a coadjoint orbit can be chosen of the 
form (z, v) £ n p with zv = 0, that is, the vector v being in the kernel of the matrix 
z. Let O be the collection of those representatives (z, v) with M(z, v) non-singular: 

O :={(z,v) en* , zv = and det M(z, v) ^ 0} . 

It is easy to see that O is the following union of SO(p)-orbits of certain "diagonal 
representatives" : 

O := SO(p)A * o„. where A = {diag(dii" 2 , ■ • ■ , d q -\h, d q ) , dj £ R*} C GL P . 

Any w £ O C n* then induces an irreducible unitary representation X w of fi and 
./V on ^(IR 9 ^ 1 )^ ~ L 2 ^- 1 ). We describe the representation of n very briefly for 
the element o n * = (z »,i> *). The construction for general w £ O can be done 
similarly by using the equivariant action of GL p on n* . As the writing of w £ O as 
GL p • o n * is not unique, there is a certain ambiguity here but it is harmless for our 
proof. 

The element o n * defines a splitting (or a complex structure) of K 2 ^ -1 - 1 = + 
Ml 9-1 . The space n is decomposed as 

(4.1) n = X P ®W = n ®t) 

where n := {z„ n , n X p ) © Rv 0n , while () := Rz Dn , + W^ 1 + R q ^ 1 is the Heisen- 
berg algebra. There exists a unique representation (A 0n » , L 2 (]R 9-1 )) of N whose 
restriction to exp Ro n « is given by the character exp i27ro n * . 
The Plancherel formula for L 2 (N) is given by 

(4.2) \\f\\l m = f \\f{w)\\ldi{w), /(0) = J^Tv(f(w))di(w) , 

where we have denoted the group Fourier transform of a function / by 

fH = [ f(g)X w (g)dg , 
Jn 

and its Hilbcrt-Schmidt norm by ||/(w)|| 2 i.e. in HS W := ^{R^ 1 ) ® L 2 (R<^ r )*. 
The Plancherel measure di can be explicitly computed but we will not need it here. 
In otherwords, the regular action of N x N on L 2 (N) is decomposed as 

(4.3) L 2 (N) = f \ w ®\* w dL(w) , 

Jo 

where A*, is the contragradient of A 2 . 

4.2. Proof of Proposition [HU Let := {{z,v)n; det(M(z,u)) ^ 0}. Clearly 
is open and dense in n, and which strictly contained in f2 C O. By elementary 
matrix computations, it can be also described as the orbit of o n » : 

Lemma 4.1. GL p acts transitively on Q and we have — GL p -o n * = GL p /Sp(q — 
1,M). 

Proof. Let (z, v) £ Q. The diagonalization of z provides a g £ SO{p) such that 
g ■ (z,v) = (w, u) with u = (ui , • • • , u p ) and 

We compute on the one hand det M(w, u) — (wi ■ ■ ■ w q -iu p ) 2 and on the other hand 
det = det M(g(z 7 v)) = det g det M(z, v) = det M(z, v) 7^ 0. Namely u p ,wi, . . . , w n / 



DEGENERATE PRINCIPAL SERIES REPRESENTATIONS OF SO(p + l,p) 11 

(A 

0. We solve now the equation g(w,u) = o n * with g G GL p of the form g = ( 
viz, 

AZiA* = J , AZ\B l + AYc = 0, cu p = l, 

which is easy to see to have a solution, e.g. by taking c — Vp 1 , B — —v~ l Z^Y 
and 

A = diag(sgnwi|wi|^, \wt\? , ■ • • ,sgpw q -i\z q -i\? , . 

(A C\ 

The isotropic subgroup in GL p of o n * consists of g = ( „ „ J € GL p with 

B = Q,C — 0, D — 1 and A such that AJ q -iA l = J q -\. Thus it is a realization of 
the symplectic group Sp(g — 1,K) and we have f2 = GL p /Sp(g — 1,R). □ 

There exists [9] a representation (r, L 2 (R 9_1 ), Mp(5 — 1, R)) of the double cover 
Mp(g - 1, R) of Sp(g - 1, R) such that 

(4.4) r(~g)X 0n , (n)r(ff)* - A „, (<? • n), .9 e Mp(q - 1,R) , 

with g G Mp(g - 1,1) H J £ Sp(g — 1,R) the double covering. Furthermore the 
representation (r,Mp(q - 1, R), i 2 (R 9_1 )) is a sum of two irreducible inequivalent 
representations [9, Theorem 4.56], 

(4.5) L 2 (R q - 1 ) = L 2 (R q - 1 )(BLj(R q - 1 ) 
of even an odd functions. 

We can now prove Proposition [ 



Proof of Provosition \3.5[ Let T be a bounded operator on L 2 (N) commuting with 
the action ir of P defined in Lemma [3T4l 

As T commutes with the left translation, by the Plancherel Theorem [7], there 
exists a measurable field {T(w),w £ O} of bounded operators on L 2 (R^ _1 ) such 
that 

(4.6) Tf{w) = f(w)f(w), f e S(N) ; 

this measurable field of operator is unique (up to a 1- negligible set). 

Let w — g * o n * with g £ SO(p)A. By the orbit method, the representations A„, 
and n <— > A 0n » (gn) are unitarily equivalent: there exists a unitary operator A w such 
that A w X w (n) — A 0n „ (gn)A w and, for / G S(N), we compute with the change of 
variable n\ — gn: 

f{w) = / f(n)A- 1 \ 0nt (gn)A w dn = A~ 1 / (n(g)) f(m)X 0nt ( ni )d ni A w 

(4.7) = A- l ^f{o n ,)A w . 

Now as T commutes with 7r((;), we obtain easily: 

ff(w) = A- 1 (T(n(g)f))~(o n *)A w . 
Using ()4.6j) and the uniqueness of {T(w), w G O}, we obtain for i-almost all w E O, 

(4.8) f(w)=A- 1 f(o n ,)A w . 

We may assume that T(o n *) exists and that relation (|4.8[) holds for all w G O. 
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In the same way, we consider g £ Mp(q — 1, M) and the equivalence relation (|4.4j) . 
Proceeding just as above, we obtain: 

(4-9) f{o n ,) = T{~g)f{o n ,)T{~g)- 1 . 

It follows then from the irreducible decomposition (|4.5[) that T(o n * ) is constant on 
each space, namely 

T(o„«) = coI + Cl U 

where U is the reflection, 

Uh{x) = h(-x) , h £ L 2 ^- 1 ) , x £ M 9_1 . 

By the Plancherel Theorem [7J, there exists a bounded operator Ti : L 2 (N) — > 
L 2 (N) which commutes with the left translations and satisfies 

(4.10) T\f(w) = f(w)A~ 1 UA w , 

for all w G O and / G S{N). Because of flU} and we have T = c J + ciT x . 

So the proof of Proposition 13.51 will be over once we have shown that c\ = and 
for this it suffices to show that Ti does not commute with the action of (71", GL P ). 

As Ti is bounded on L 2 {N) and commutes with left translation, it is a convolu- 
tion operator with a tempered kernel: there exists k G S'{N) such that T\f = f * re 
for any / G S(N). We claim that re is not invariant under GL p and this shows that 
T\ does not commute with the action of (tt, GL p ). 

To show our claim, we first compute Ti/(0) for / G S(N). For this we will use 
the Plancherel formula (see (|4.2jl ): 

(4.H) Ti/(0) = / Tr(Ti/H) . 

Jo 

Now by (|4.7[) and (|4. 10[) , for w = g * o n * , we have: 

(4-12)' 

Tr (tT/h) = Tr (J^A^UaJ) = Tr (A w f(w)A^ 1 V > j = Tr (n(g)fMu) . 

So we just want to compute the trace of f(o n *)U on the Hilbert space L 2 (R 9_1 ). 
This can be derived from the standard formulas f |23l Chapt. XII, §6], [551 Chapt. 
II, §2-§3] ) for the Weyl transform. 

Indeed considering the decomposition (|4.ip , we write the elements of n as h + h 1 - 
where h € t) and h 1 - G rio- Integrating / over no, we obtain the function F with f): 

F(h) = { fih + h^dh- 1 . 

J no 

We now identify f) with the Heisenberg group and we write the elements of (] 
as h = (x,y,t) £ R 9-1 x R 9_1 x M. It is clear that /(o n ») coincides with the 
Schrodinger representation of the Heisenberg group at F. From [23l Chapt. XII, 
§6.3], this shows that /(o n ») is the integral operator on L 2 (R 9_1 ) with kernel 

Kf,(x,y) := c [ e 2 ^ u < y+x)+t) F(u,y - x,t)dudt , x,y£R q -\ 

where c = c q is a known constant (our £ here corresponds to jt in [231 Chapt. XII, 
§6.3, (91)]). So the kernel of the operator f(o n *)U is K^(—x,y) and we can now 
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compute the trace of the operator: 

Tr (f(o a .)u) = / K h {-x,x)dx 

= c[ I e 27rit F{u,2x,t)dudtdx . 

Thus we have obtained: 

Tr (f(o n ,)u)=C(Ff)(o n *) , 

for some non-zero known constant C where we have denoted by J- the Euclidean 
Fourier transform on n, that is, 

Tf{(,v) = [ f(z,v)e 2i7r ^ + ^dzdv , 
J n 

where we have used the canonical Euclidean scalar products on X q ^\ and R 9-1 . 
Now by (|4.12|) . this shows that for any w £ O, we have: 

Tr (J(w)u) =C(J-/)H . 

By (|4.1ip . we obtain: 

J f{n)K{n- l )dn = Ti/(0) =C J T(w)di{w) . 

Hence the support of J-7c(- -1 ) is included in O. But O is invariant under n i— > n^ 1 
but not invariant under GL p since O is strictly included in = GL p ■ o n « (see 
Lemma f4 . ljl . So k is not GL p -invariant. This shows that T± does not commute 
with (n, GL P ) and concludes the proof of Proposition E3I □ 

4.3. Decomposition of L 2 (N) under N x Sp(g — 1,R). We note that the above 
proof also yields a decomposition of L 2 (N) under the action of N x Sp(g — 1,R). 
Consider first the reference point w — o n « and the space L 2 (R 9_1 )(X)L 2 (R 9_1 ) with 
N acting on the left factor by X w . Note that the metaplectic representation r on 
i 2 (M 9_1 ), by its definition, defines a unitary representation of t®t* of Sp(g — 1, R) 
on L 2 (R 9_1 ) (8) i 2 (R g_1 ), viewed as Hilbert-Schmidt operators, by 

{T®T*)(g)T = T{g)TT*{g) . 

The group N x Sp(q - 1,R) acts on L 2 (R«~ 1 ) <g> i 2 ^" 1 ) and we denote the 
corresponding representation, by \ u x t u . 

Using the decomposition of L 2 (R 9 ~ 1 ) into even and odd functions L 2 (K. q ~ 1 )i, 
i = 0, 1, we have: 

L 2 (R q ^ 1 ) (g> L 2 (R g_1 ) = L 2 ^" 1 ) ® L a (K«- 1 )o © L 2 (R 9_1 ) ® L 2 (R ?_1 ) 1 , 
and we obtain the TV x Sp(g — 1, R)-irreducible decomposition: 

A w x t w = (A w x r w ) + (A w x r w )i . 
Clearly this construction can be done for any w. We have then 
Corollary 4.2. 77ie space L 2 (N) is decomposed under N x Sp(g — 1,R) as 

L 2 (N) = / (X u X r w ) + (A w x r w )i di(to) . 
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